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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)K Responsive to communication(s) filed on 24 February 2004 . 
2a)E3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-20 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) E3 Claim(s) 7-10 is/are allowed. 

6) E3 Claim(s) 1-5.11-14 and 16-20 is/are rejected. 

7) M Claim(s) 6 and 15 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) 13 The drawing(s) filed on 12 February 2002 is/are: a)E3 accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 



Response to Amendment 



1 . In response to the office action from 1 1/24/2004, the applicant has submitted an 
amendment, filed 2/24/2005, arguing to traverse the art rejection based on the alleged use of 
hindsight reasoning (Amendment, Pages 3-4). Applicant's arguments have been folly considered, 
however the previous rejection is maintained due to the reasons listed below in the response to 
arguments. 

2. The applicant's arguments with respect to the draftperson's objection (PTO 948) 
(Amendment, Page 2) are convincing. Thus, the draftperson's objection is withdrawn and the 
formal drawings from 2/12/2002 are accepted. 

Response to Arguments 

3. Applicant's arguments have been fully considered but they are not persuasive for the 
following reasons: 

With respect to Claim 1, the applicant argues that there is a lack of motivation to 
combine Lange et al (U.S. Pub: 2001/0025241), Boll ("Suppression of Acoustic Noise in Speech 
Using Spectral Subtraction, " 1979), and Hauptmann et al. ("Text, Speech, and Vision for Video 
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Segmentation: The Informedia Project, " J 995) and that the examiner has used hindsight reasons 
in combining these references (Amendment, Pages 3-4). 

In response to applicant's argument that the examiner's conclusion of obviousness is 
based upon improper hindsight reasoning, it must be recognized that any judgment on 
obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so 
long as it takes into account only knowledge which was within the level of ordinary skill at the 
time the claimed invention was made, and does not include knowledge gleaned only from the 
applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 
170 USPQ 209 (CCPA 1971). 

In response to applicants argument that there is no suggestion to combine the references, 
the examiner recognizes that obviousness can only be established by combining or modifying the 
teachings of the prior art to produce the claimed invention where there is some teaching, 
suggestion, or motivation to do so found either in the references themselves or in the knowledge 
generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 
USPQ2d 1596 (Fed. Cir. 1988) mdln re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 
1992). 

In this case, Lange and Hauptmann are related ait in that they both utilize speech 
recognition in developing a transcript of an audio signal associated with a video signal. As 
required by the MPEP (2143.01), the motivation for combining the prior art is found in 
Hauptmann, as is noted in the previous office action (Page 5), and specifically relates to the 
benefit of utilizing included caption data when available since it is less prone to error than 



Application/Control Number: 09/820,401 
Art Unit: 2655 



Page 4 



caption data that would be obtained using speech recognition (Hauptmann, Page 5, Section 4, 
Paragraph 2). 

Boll is also related art since it is from a similar field of endeavor in speech recognition or 
creating a transcript of a speech input and provides the teaching of a well-known and widely 
used method of spectral subtraction. Spectral subtraction is commonly used in speech 
recognition for the benefit of increased speech recognition accuracy by improving speech 
intelligibility, which is noted by Boll and cited in the prior office action as the reason for 
combining the prior art (prior office action, Page 5; and Boll, Summary and Conclusion). 

Thus, since the motivation for combining the references is given in the references 
themselves, the reference combination is proper. 

Claims 11 and 16 contain subject matter similar to Claim 1, and thus, also remain 
rejected for the above noted reasons. 

The dependent claims further limit rejected independent claims (1, 11, and 16), and thus, 
also remain rejected. 

Claim Rejections - 35 USC §103 

4. The following is a quotation of 35 U.S. C 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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5. Claims 1, 2, 4, 16, 17, and 19 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Lange et al (U.S. Pub: 2001/0025241) in view of Boll ("Suppression of Acoustic Noise in 
Speech Using Spectral Subtraction, " 1979), and further in view of Hauptmann et al. ("Text, 
Speech, and Vision for Video Segmentation: The Informedia Project, " 1995) 
With respect to Claims 1 and 16, Lange discloses: 

A method and computer readable carrier (carrier defined as a magnetic or optical disk as 
per paragraph 43 in the application specification) (conversion of an audio component ofanAV 
signal into captions implemented using a computer readable medium containing a program, 
Paragraph 16, Lines 6-10) including computer program instructions, respectively, for displaying 
text information corresponding to a speech portion of audio signals of a television program to as 
a closed caption on an video display device (AV captioning system including a display device, 
Fig. 1 Element 60, and a speech-to-text processor, Paragraph 16, Lines 3-6, also, the display 
device can be a television capable of processing closed captioning data, Paragraph 23, Lines 1- 
5), the method comprising the steps of: 

Decoding the audio signals of the television program (signal separation processing 
system, Fig, 1, Element 30, for separating the audio signal from an A V signal, Paragraph 19, 
Lines 1-7)', 

Parsing the speech portion into discrete speech components in accordance with a speech 
model and grouping the parsed speech components (vocabulary used for recognizing speech as 
word units, Paragraph 31, Lines 5-7); 
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Identifying words in a database corresponding to the grouped speech components 
(vocabulary database used for word recognition within a speech component of an audio signal, 
Paragraph 31, Lines 5-7)\ and 

Converting the identified words into text data for display on the display device as the 
closed caption (text box area for displaying text as it is transcribed, Paragraph 31, Lines 24-25). 

Lange does not teach the filtering of audio signals by using a spectral subtraction method, 
however, Boll discloses: 

Filtering the audio signals to extract the speech portion (filtering noise from a speech 
signal through a spectral subtraction noise suppression method, Abstract). 

Neither Lange nor Boll teaches the additional step of determining the presence of and 
using closed captioned data and if no closed caption data is present, performing speech 
recognition in order to obtain closed caption text, however Hauptmann discloses such a method 
(inherent step of closed caption transcript detection to determine if a closed-caption is present 
and, if not, utilizing speech recognition to produce closed caption text, Page 5, Section 4, 
Paragraph 2). 

Lange, Boll, and Hauptmann are analogous art because they are from a similar field of 
endeavor in speech processing for recognition. Thus, it would have been obvious to a person of 
ordinary skill in the art, at the time of invention, to combine the use of spectral subtraction to 
suppress noise in a speech signal as taught by Boll with the method for converting the audio 
portion of an AV signal into captions using a speech recognition and speech-to-text conversion 
means as taught by Lange to provide more accurate speech recognition due to increased speech 
intelligibility (Boll, Summary and Conclusions) by enhancing a speech portion of an audio signal 
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through spectral subtraction. Also, it would have been further obvious to one of ordinary skill in 
the art at the time of invention, to modify the teachings of Lange in view of Boll with the method 
for caption detection and speech recognition usage only if a closed caption signal is not present 
as taught by Hauptmann for the benefit of providing a more accurate television program 
transcription by only performing speech recognition if necessary since closed caption data is less 
prone to error (Hauptmann, Page 5, Section 4, Paragraph 2). 

With respect to Claims 2 and 17, Lange additionally discloses: 
A method and computer readable carrier, according to claims 1 and 16, respectively, 
wherein the step of filtering the audio signals is performed concurrently with the inherently 
concurrent step of decoding of later-occurring audio signals of the television program and step of 
parsing of earlier occurring speech signals of the television program (captioning of an A V signal 
as it is received, Paragraph 27). 

With respect to Claims 4 and 19, Lange further recites: 

A method and computer readable carrier according to claims 1 and 16, respectively, 
further including the step of formatting the text data into lines of text data for display in a closed 
caption area of the display device (suggested option of caption formatting in a two or three-line 
roll-up format, Paragraph 23, Lines 17-25). 

6. Claims 3 and 18 are rejected under 35 U.S.C. 103(a) as being unpatentable over Lange 
in view of Boll, in further view of Hauptmann et al, and in further view of Ortega et al (U.S. 
Patent: 6,332,122). 

With respect to Claims 3 and 18: 

\ 
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Lange in view of Boll, and further in view of Hauptmann teaches the method and 
computer program for converting the audio portion of an AV signal into captions that recognizes 
individual words within speech data by using a vocabulary database, utilizes spectral subtraction 
pre-processing, and features caption detection, as applied to Claims 1 and 16, however Lange in 
view of Boll, and further in view of Hauptmann does not teach: the use of a speaker independent 
model in individual word recognition as recited in Claims 3 and 18. 

Ortega discloses: 

A method and computer readable medium according to claims 1 and 16, respectively, 
wherein the step of parsing the speech portion into discrete speech components includes the step 
of employing a speaker independent model to provide individual words as the parsed speech 
components (speech frames identified through the use of a speaker independent model, Col 4, 
Lines 35-48). 

Lange, Boll, Hauptmann, and Ortega are analogous art because they are from a similar 
field of endeavor in speech processing for recognition. Thus, it would have been obvious to a 
person of ordinary skill in the art, at the time of invention, to combine the use of a speaker 
independent model in speech recognition as taught by Ortega with the teachings of Lange in 
view of Boll, and further in view of Hauptmann to create a captioning method in which the 
speaker can be easily identified by a viewer in the case where the system has not been trained for 
that particular speaker. Therefore, it would have been obvious to combine Ortega with Lange in 
view of Boll, and further in view of Hauptmann for the benefit of obtaining a captioning method, 
utilizing spectral subtraction pre-processing and caption detection, capable of identifying a 
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spoken utterance even if the speech recognition system has not been trained with that speaker's 
voice, to obtain the invention as specified in Claims 3 and 18. 

7. Claims 5, 11, 12, 14, and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Lange in view of Boll, in further view of Hauptmann et al and in further view of Ditzik 
(US, Patent: 6,415,256). 

With respect to Claims 5 and 20: 

Lange in view of Boll, and further in view of Hauptmann teaches the method and 
computer program for converting the audio portion of an AV signal into captions that recognizes 
individual words within speech data by using a vocabulary database, utilizes spectral subtraction 
pre-processing, and features caption detection, as applied to Claims 1 and 16, however Lange in 
view of Boll, and further in view of Hauptmann does not teach: a speaker dependent model for 
providing individual phonemes extracted from the speech signal as recited in Claim 5. 

Ditzik suggests: 

A method and computer readable carrier according to claims 1 and 16, respectively, 
wherein the step of parsing the speech portion into discrete speech components includes the step 
of employing a speaker dependent model to provide phonemes as the parsed speech components 
(speech recognition system designed for speaker dependent or speaker independent operation, 
Col 3, Lines 1 7-19 and phoneme modeling database used for speech recognition that may 
contain special or generalized vocabularies, Col 3, Lines 52-59). 

Lange, Boll, Hauptmann, and Ditzik are analogous art because they are from a similar 
field of endeavor in speech processing for recognition. Thus, it would have been obvious to a 
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person of ordinary skill in the art, at the time of invention, to combine the use of a speaker 
dependent model in speech recognition to provide phonemes from speech data as taught by 
Ditzik with the teachings of Lange in view of Boll, and further in view of Hauptmann to create a 
captioning method in which the speaker can be easily identified by a viewer in the case where 
the system has been trained for that particular speaker and in which a best word choice can be 
made on an individual phoneme basis. Therefore, it would have been obvious to combine Ditzik 
with Lange in view of Boll, and further in view of Hauptmann for the benefit of obtaining a 
captioning method, utilizing spectral subtraction pre-processing and caption detection, capable of 
identifying a speaker that has been previously trained within a speech recognition system 
featuring a more accurate word identification method on a phoneme-by-phoneme basis, to obtain 
the invention as specified in Claims 5 and 20. 
With respect to Claim 11: 

Lange in view of Boll, and further in view of Hauptmann teaches the method and 
computer program for converting the audio portion of an AV signal into captions that recognizes 
individual words within speech data by using a vocabulary database, utilizes spectral subtraction 
pre-processing, and features caption detection, as applied to Claims 1 and 16, however Lange in 
view of Boll, and further in view of Hauptmann does not teach: a phoneme identification as 
recited in Claim 11. 

Ditzik discloses: 

A phoneme generator, which parses the speech portion into phonemes in accordance with 
a speech model (speech processing utilizing a phoneme modeling function, which divides speech 
into phonemes using phoneme modeling and HMM algorithms, Col. 3, Lines 30-43)\ 
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A database of words, each word being identified as corresponding to a discrete set of 
phonemes (phoneme modeling database that contains a grammar rules database for discerning 
allowable word sequences, word dictionaries, and a phonological rules database for providing 
data to the phoneme modeling program, Col. 3, Lines 52-67) ; and 

A word matcher which groups the phonemes provided by the phoneme generator and 
identifies words in the database corresponding to the grouped phonemes (word dictionary 
suggests word recognition from input phonemes, and inherently, phonemes would also be 
grouped to form a word since words can consists of multiple phonemes. Phoneme groupings in 
word form are then further grouped in a word sequence using a grammar dictionary, Col. 3, 
Lines 55-59). 

Lange, Boll, Hauptmann and Ditzik are analogous art because they are from a similar 
field of endeavor in speech processing for recognition. Thus, it would have been obvious to a 
person of ordinary skill in the art, at the time of invention, to combine a means of separating 
input speech into phonemes for word recognition via a database as taught by Ditzik with the 
teachings of Lange in view of Boll, and further in view of Hauptmann to create a captioning 
method in which a best word choice can be made on an individual phoneme basis through 
phoneme grouping and utilizing a known word database for efficient processing. Therefore, it 
would have been obvious to combine Ditzik with Lange in view of Boll, and further in view of 
Hauptmann for the benefit of obtaining a captioning method, utilizing spectral subtraction pre- 
processing, and featuring a more efficient word identification method on a phoneme-by-phoneme 
basis by examining only the best word choices within a database, to obtain the invention as 
specified in Claim 1 1 . 
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With respect to Claim 12: 

Lange in view of Boll, and further in view of Hauptmann teaches captioning, utilizing 
spectral subtraction pre-processing, of an AV signal in real-time, which is indicative of parallel 
processing as applied to Claim 2, however Lange in view of Boll, and further in view of 
Hauptmann does not teach a phoneme generator as recited in Claim 12. 

Ditzik discloses the phoneme modeling function as applied to Claim 11. 

Lange, Boll, Hauptmann and Ditzik are analogous art because they are from a similar 
field of endeavor in speech processing for recognition. Thus, it would have been obvious to a 
person of ordinary skill in the art, at the time of invention, to combine a means of separating 
input speech into phonemes for word recognition via a database as taught by Ditzik with the 
teachings of Lange in view of Boll, and further in view of Hauptmann to create a captioning 
method in which a best word choice can be made on an individual phoneme basis through 
phoneme grouping and utilizing a known word database for efficient processing. Therefore, it 
would have been obvious to combine Ditzik with Lange in view of Boll, and further in view of 
Hauptmann for the benefit of obtaining a real-time captioning method, utilizing spectral 
subtraction pre-processing and caption detection, and featuring a more efficient word 
identification method on a phoneme-by-phoneme basis by examining only the best word choices 
within a database, to obtain the invention as specified in Claim 12. 

With respect to Claim 14, 

Lange in view of Boll, and further in view of Hauptmann teaches the method, system, 
and computer program for converting the audio portion of an AV signal into captions as applied 
to Claim 1, however Lange in view of Boll, and further in view of Hauptmann does not teach a 
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speaker dependent speech recognition system containing a phoneme generator as recited in 
Claim 14. 

Ditzik discloses speaker dependent speech recognition as applied to Claim 5 and a 
phoneme modeling function as applied to Claim 1 1 . 

Lange, Boll, Hauptmann, and Ditzik are analogous art because they are from a similar 
field of endeavor in speech processing for recognition. Thus, it would have been obvious to a 
person of ordinary skill in the art, at the time of invention, to combine the use of a speaker 
dependent model in speaker dependent speech recognition to provide phonemes from speech 
data as taught by Ditzik with the teachings of Lange in view of Boll, and further in view of 
Hauptmann to create a captioning method in which the speaker can be easily identified by a 
viewer in the case where the system has been trained for that particular speaker and in which a 
best word choice can be made on an individual phoneme basis. Therefore, it would have been 
obvious to combine Ditzik with Lange in view of Boll, and further in view of Hauptmann for the 
benefit of obtaining a captioning method capable of identifying a speaker that has been 
previously trained within a speech recognition system featuring a more accurate word 
identification method on a phoneme-by-phoneme basis, to obtain the invention as specified in 
Claim 14. 

8. Claim 13 is rejected under 35 U.S.C. 103(a) as being unpatentable over Lange in view of 
Boll, in further view of Hauptmann, in further view of Ditzik, and in yet further view of Ortega. 
With respect to Claim 13: 
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Lang in view of Boll, in further view of Hauptmann and in further view of Ditzik teaches 
the captioning system and computer program utilizing a vocabulary for word identification from 
phoneme groupings and spectral subtraction pre-processing, as applied to Claim 1 1, but does not 
teach the use of a speaker independent model as recited in Claim 13. 

Ortega recites the use of a speaker independent model in speech frame recognition as 
applied to Claim 3. 

Lange, Boll, Hauptmann, Ditzik, and Ortega are analogous art because they are from a 
similar field of endeavor in speech processing for recognition. Thus, it would have been obvious 
to a person of ordinary skill in the art, at the time of invention, to combine the use of a speaker 
independent model in speech recognition as taught by Ortega with the teachings of Lang in view 
of Boll, in further view of Hauptmann and in further view of Ditzik to create a captioning 
method in which the speaker can be easily identified by a viewer in the case where the system 
has not been trained for that particular speaker and in which a best word choice can be made on 
an individual phoneme basis through phoneme grouping utilizing a known word database for 
. efficient processing. Therefore, it would have been obvious to combine Ortega with Lang in 
view of Boll, in further view of Hauptmann, and in further view of Ditzik for the benefit of 
obtaining a more flexible captioning method, utilizing spectral subtraction pre-processing and 
caption detection, capable of identifying speech by a speaker even if the speech recognition 
system has not been trained with that speaker's voice, to obtain the invention as specified in 
Claim 13. 
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Allowable Subject Matter 

9. Claims 6 and 15 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 

10. Claims 7-10 are allowed. 

1 1 The following is a statement of reasons for the indication of allowable subject matter: 
Prior art teaches the captioning system that converts the audio portion of an AV signal 
through individual phoneme recognition and word sequencing and features speaker 
dependent/independent recognition, decoding, filtering, and processing means as applied to 
Claims 1-5, 1 1-14, and 16-20. Additionally, "An Automatic Caption-superimposing System 
with a New Continuous Speech Recognizer" by Imai, Ando, and Miyasaka teaches a captioning 
system utilizing speech recognition training for error minimization performed before television 
signal transmission. 

The prior art of record does not teach: 

• A transmitted television signal containing training text used for updating a HMM 
to be used in phoneme identification as recited in Claims 6 and 15. 

• A transmitted television signal containing training text used for generating a 
HMM as recited in Claim 7. 

Claims 8, 9, and 10 further limit their parent claims, and thus, are allowable. 
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Conclusion 

12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1. 136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

13. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to James S. Wozniak whose telephone number is (571) 272-7632 
and email is James.Wozniak@uspto.gov. The examiner can normally be reached on Mondays- 
Fridays, 8:30-4:30. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Wayne Young can be reached at (571) 272-7582. The fax/phone number for the 
Technology Center 2600 where this application is assigned is (703) 872-9306. 
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Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the technology center receptionist whose telephone number is (703) 306- 
0377. 



James S. Wozniak 
6/6/2005 




